Natural Interactivity Resources - Data, Annotation Schemes and Tools

This paper presents results of three surveys of natural interactivity and multimodal resources carried out by a Working Group in the ISLE project on International Standards for Language Engineering. Information has been collected on a large number of corpora, coding schemes and coding tools world-wide. The paper presents the information collection process, the description and validation methods used, the surveyed resources, and brief conclusions for each of the three resource areas reviewed. Observations on user profiles, user needs and best practices are briefly presented. 1. International Standards for Language Engineering surveys The long-term vision of natural interactivity envisions that humans communicate, or exchange information, with machines (or systems) in the same ways in which humans communicate with one another, using thoroughly coordinated speech, gesture, gaze, facial expression, head movement, bodily posture, and object manipulation [Bernsen 2001]. The idea of multimodality is to improve human-system interaction in various ways by using novel combinations of (unimodal) input/output modalities [Bernsen 2002]. Natural interactivity is by nature (mostly) multimodal. Across the world, researchers and companies are beginning to focus on investigating and exploiting the potential of natural interactive and multimodal systems. An important foundation for work on such systems is resources, i.e. data (corpora), corpus annotation schemes and annotation tools. A good starting-point for those working in the field therefore is information about which resources are already there, how they might be accessed, what they might be used for, etc., so that fewer people try to re-invent the wheel than might otherwise be the case. This paper presents substantial and, to our knowledge, unprecedented groundwork on resources carried out in the European Natural Interaction and Multimodality (NIMM) Working Group of the EU-HLT/US-NSF project International Standards for Language Engineering (ISLE). ISLE is the successor of EAGLES (European Advisory Group for Language Engineering Standards) I and II and includes working groups on lexicons, machine translation evaluation, and NIMM, respectively. The NIMM working group (isle.nis.sdu.dk) began its work in early 2000 and has now completed three comprehensive surveys. The surveys address NIMM data, annotation schemes, and annotation tools, respectively. Focus has been on producing resource descriptions which are systematic, follow standard formats, and are sufficient for providing interested parties in research and industry with the information they need to decide if a particular resource matches their interests. Each resource comes with contact information on its creator(s). The three surveys are available in html and pdf format at the ISLE NIMM website isle.nis.sdu.dk. The report on NIMM data resources [Knudsen et al. 2002a] reviews a total of 64 resources world-wide, 36 of which are facial resources and 28 are gesture resources. Several corpora combine speech with facial expression and/or gesture. The report also includes a survey of market and user needs produced by ELRA (the European Language Resources Agency) and 28 filled questionnaires collected at the Dagstuhl workshop on Coordination and Fusion in Multimodal Interaction held in late 2001. The survey of NIMM corpus annotation schemes [Knudsen et al. 2002b] reviews 7 descriptions of coding schemes for facial expression and speech, and 11 descriptions of annotation schemes for gesture and speech. The survey of NIMM corpus coding tools [Dybkjær et al. 2001] describes 12 annotation tools and tool projects most of which support speech annotation combined with gesture annotation, facial expression annotation, or both. In the following, we first describe the three surveys in more detail (Sections 2-4). We then present conclusions on users and resources (Section 5).